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Method and System for Pseudo-Autonomous Image Registration 

FIELD AND BACKGROUND OF THE INVENTION 

The present invention relates to image correlation and, in particular, it 
concerns a platform with pseudo-autonomous image registration. 

■ "5 * " It is known to perform image registration between images viewed at 

different viewing angles. One example of particular interest is registration of a 
perspective image with an orthogonal photographic representation which 
allows features in the perspective image to be associated with their geographic 
locations. The techniques for performing such registration per se are well 

10 known in the art and will not be de^lt with here in detail. 

The following patent documents, mentioned as illustrative of 
applications of image registration techniques, are hereby incorporated by 
reference for all purposes as if fully set forth herein. Israeli Patent No. 1 1 1069 
to Sroka et al. describes a method and system for marking object images 

15 acquired at high zooming by an airborne imaging device on a reference image 
having a wide field of view of the scene. U.S. Patent Application Publication 
No. 2002/0196248 to Kraus et al. describes a method for sharing visual 
information beUveen a number of command and control ("C&C") base units. 
Co-pending, co-assigned Israeli Patent Application No. 158685, which is 

20 unpublished at the date of filing this application and does not constitute prior 
art, extends these concepts to bi-directional communication between platforms 
via geographic coordinates which are represented in the local perspective view. 
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Turning now to Figure 1, this shows a typical example of a scheme for 
implementing image correlation for an airborne platform with an imaging 
system, for example an unmanned aerial vehicle ("UAV") 16. An image of a 
region of interest is derived from a camera having an imaging sensor 10 
5 operated by- a camera controller unit 14 and is transmitted via wireless 
transceivers 20, 22 to a base station 18, where it is processed by a processing 
system 24 and may be presented on a display 12 located at the base station or 
any other desired location. Camera controller unit 14 typically operates an 
optical zoom of imaging sensor 10 for selectively generating views with 

10 relatively wide field of view (FOV) and relatively narrow FOV. A "frozen" 
wide angle view is optionally displayed on a second display 26, preferably with 
a frame indicating the region of the zoomed view currently being displayed on 
display 12 to facilitate user orientation. A similar system and various additional 
preferred features for operation of the system are described in the 

15 aforementioned Israeli Patent No. 1 1 1069. 

Each base unit 18 is also provided with reference data 30 corresponding 
to an orthogonal photographic representation of a region overlapping at least 
part of the field of view of the perspective view of imaging sensor 10. The 
orthogonal photographic representation is preferably associated with a defined 

20 geographic coordinate system, typically corresponding to the standard 
longitude- latitude or "north-south" coordinate system defined globally across 
the surface of the Earth. Finally in structural terms, the base unit 18 is typically 
in communication with other base units or other entities via a wireless data 
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communication system, such as a data-link system, represented by transceiver 
32. 

Figure 2 illustrates the relationship between the images/views received 
by the base unit 18. Specifically, there are shown a real-time detailed narrow 
5 FOV perspective image 40, a frozen wide-angle perspective image 42 and an 
orthogonal photographic representation 44. The wide-angle image is typically 
necessary to provide sufficient data for reliable correlation with the orthogonal 
photographic representation. As a result, correlation between the perspective 
view and the orthogonal photographic representation is performed as a two-step 

10 correlation as illustrated in Figure 3. Specifically, the wide FOV image 42 is 
correlated with the orthogonal photographic representation 44 to generate a 
first mapping transformation T 1? and the narrow FOV image 40 is correlated 
with the wide FOV image 42 to generate a second mapping transformation T 2 . 
The combination of mapping transformations Ti and T 2 or their inverses 

15 together fully define the mapping between the perspective image 40 and the 
orthogonal photographic representation 44 in both directions, for example, 
allowing a target 46 identified in perspective view 40 to be associated with a 
geographic coordinate in orthogonal representation 44 or the reverse. 

It should be noted that the entire image correlation processing is 

20 performed at base unit 18 remote from the data gathering platform 16. This is 
necessitated by the heavy data processing and huge quantities of data typically 
involved in image registration of this type, rendering it impractical to perform 
the processing locally for each platform. Specifically, the data storage and 
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processing requirements of correlating images to a geographical database 
typically require a server system with multiple parallel processors and 
corresponding instant-access mass-data storage systems. These requirements 
render it impractical fqr all but the most massive mobile platforms to carry a 
5 self-contained geographic-database image registration system. Furthermore, the 
need for continual updating of information in the geographical database 
encourages a centralized processing approach. 

As a result of this centralized processing approach, each mobile platform 
is required to continuously download sampled images to the remote base unit 
10 site. This occupies a disproportionate amount of communication bandwidth, 
and renders the mobile platform completely dependent upon the base unit. The 
continuous computational load on the base unit also limits the number of such 
platforms which can be served by a given base unit. 

There is therefore a need for a method and system which would provide 
15 pseudo-autonomous image correlation for a platform-based system. 

SUMMARY OF THE INVENTION 

The present invention is a system and method for providing a platform 
with pseudo-autonomous correlation between a perspective view generated by 
a platform-based imaging sensor and an orthogonal photographic 
20 representation. 

According to the teachings of the present invention there is provided, a 
method for providing a platform with pseudo-autonomous correlation between 
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a perspective view generated by a platform-based imaging sensor and an 
orthogonal photographic representation, the method comprising: (a) storing on 
the platform: (i) a reference perspective view at least partially overlapping a 
current field of view of the imaging sensor, and (ii) parameters of a primary 
mapping transformation corresponding to a mapping between the orthogonal 
photographic representation and the reference perspective image; (b) obtaining 
a current perspective view from the imaging sensor; and (c) employing a 
processing system located on the platform to: (i) correlate the current 
perspective view with the reference perspective view so as to generate a 
secondary mapping transformation between the current perspective view and 
the reference perspective view, and (ii) use the primary and secondary mapping 
transformations to map equivalent points in at least one direction between the 
current perspective view and the orthogonal photographic representation. 

According to a further feature of the present invention, the current 
perspective view has a field of view corresponding to less than ten percent of 
an area of a field of view of the reference perspective view. 

According to a further feature of the present invention, the reference 
perspective view is derived from the imaging sensor, and wherein the 
parameters of a primary mapping transformation are derived by: 
(a) downloading the reference perspective view from the platform to a remote 
processing system; (b) employing the remote processing system to correlate the 
reference perspective view with the orthogonal photographic representation so 
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as . to generate the parameters; and (c) uploading the parameters ' from the 
remote processing system to the platform. 

According to a further feature of the present invention, a mismatch 
criterion between the current perspective view and the reference perspective 
5 view is evaluated; and conditional upon a result of the mismatch criterion 
evaluation, the imaging sensor is employed to generate a new reference 
perspective view and repeating the steps of: downloading; employing the 
remote processing system; and uploading. 

According to a further feature of the present invention, wherein the 
10 platform is an airborne platform. 

According to a further feature of the present invention, the orthogonal 
photographic representation has a defined geographic coordinate system. 

According to a further feature of the present invention, the reference 
perspective view is derived from a source other than the imaging sensor, the 
1 5 platform being an airborne platform, the step of storing being performed prior 
to launch of the platform. 

According to a further feature of the present invention, the storing 
includes storing a plurality of the reference perspective views and a plurality of 
sets of corresponding parameters of the primary mapping transformation, the 
20 method further comprising selecting one of the plurality of the reference 
perspective views at least partially overlapping the current field of view of the 
imaging sensor for correlating with the current perspective view. 
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There is also provided according to the teachings of the present 
invention, an image correlation system for providing pseudo-autonomous 
platform-based correlation between a perspective view generated by a 
platform-based imaging sensor and an orthogonal photographic representation, 
5 the system comprising: (a) a platform-based imaging sensor for generating 
perspective views; (b) a platform-based data storage device for storing: (i) a 
reference perspective view, and (ii) parameters of a primary mapping 
transformation corresponding to a mapping between the orthogonal 
photographic representation and the reference perspective image; and (c) a 

10 platform-based processing system associated with the imaging sensor and with 
the data storage device, the processing system being configured for: 
(i) correlating a current perspective view derived from the imaging sensor with 
the reference perspective view so as to generate a secondary mapping 
transformation between the current perspective view and the reference 

1 5 perspective view, and (ii) employing the primary and secondary mapping 
transformations to map equivalent points in at least one direction between the 
current perspective view and the orthogonal photographic representation. 

According to a further feature of the present invention, there is also 
provided: (a) a remote processing system including a remote data storage 

20 device storing the orthogonal photographic representation; and (b) a 
communication link between the platform-based processing system and the 
remote processing system, wherein the remote processing system is configured 
to: receive an.^image downloaded from the imaging sensor via the 
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communication link, the image being designated as a reference perspective 
view; correlate the reference perspective view with the orthogonal 
photographic representation so as to generate the parameters; and upload the 
parameters via the communication link to the platform-based data storage 
5 device. 

According to a further feature of the present invention, the 
communication link is a wireless communication link. 

According to a farther feature of the present invention, the platform is an 
airborne platform. 

10 According to a farther feature of the present invention, the orthogonal 

photographic representation has a defined geographic coordinate system. 

According to a further feature of the present invention, the platform- 
based data storage device is configured to store a plurality of the reference 
perspective views and a plurality of sets of corresponding parameters of the 

15 primary mapping transformation, the platform-based processing system being 
further configured to select one of the plurality of the reference perspective 
views for correlating with the current perspective view. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of example only, with 
20 reference to the accompanying drawings, wherein: 
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FIG. 1 is a schematic representation of a conventional system for 
providing a user with correlation between a perspective view generated by a 
platform-based imaging sensor and an orthogonal photographic representation; 
FIG. 2 is a schematic representation of the images used by the system of 
5 Figure 1; 

FIG. 3 is a schematic representation of a two-stage correlation 
performed by the system of Figure 1; 

FIG. 4 is a schematic representation of a system, constructed and 
operative according to the teachings of the present invention, for providing a 
10 platform with pseudo-autonomous correlation between a perspective view 
generated by a platform-based imaging sensor and an orthogonal photographic 
representation; 

FIG. 5 is a flow diagram illustrating the basic operation of the system of 
Figure 4 and the corresponding method according to the teachings of the 
1 5 present invention; and 

FIG. 6 is a flow diagram illustrating a preferred implementation of the 
method of Figure 5. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention is a system and method for providing a platform 
20 with pseudo-autonomous correlation between a perspective view generated by 
a platform-based imaging sensor and an orthogonal photographic 
representation. 

9 
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The principles and operation of systems and methods according to the 
present invention may be better understood with reference to the drawings and 
the accompanying description. 

By way of introduction, the fundamental concept underlying the present 
5 invention is the observation that the two-stage correlation calculation illustrated 
in Figure 3 can be subdivided between two locations. Specifically, the 
calculation of the primary mapping transformation between the orthogonal 
photographic representation and the wide FOV reference image, which 
involves very large quantities of data and imposes a heavy processing load, can 

10 be performed at a remote location to determine the parameters of the mapping. 
This mapping remains constant for as long as the given reference image is 
being used. The secondary mapping, on the other hand, between the 
perspective reference view 42 and the current perspective view 40 involves 
very much less data and a much smaller processing load, but must be updated 

15 for each new camera view. By providing a platform with the primary mapping 
parameters calculated at a remote location and by using a platform-based 
processing system to derive the secondary mapping, the platform is provided 
with the ability to autonomously correlate newly acquired perspective views 
with a coordinate system based on the orthogonal photographic representation. 

20 This in turn allows locations within the newly acquired images to be related 
directly to the geographical coordinate system, allowing for concise and precise 
coordinate-based communication with the base unit and other platforms for 
purposes of tracking, detection, orientation and navigation. By facilitating bi- 

10 
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directional communication using N-S coordinates to designate locations of 
interest, communication can be achieved is a very efficient compact data 
format at high speeds with very narrow bandwidth usage. 

Referring now » to the drawings, Figures 4-6 show a preferred 
5 implementation of an image correlation system 100, constructed and operative 
according to the teachings of the present invention, and the corresponding 
method, for providing pseudo-autonomous platform-based correlation between 
a perspective view generated by a platform-based imaging sensor and an 
orthogonal photographic representation. System 100 is illustrated here as a 
10 modification of the system of Figure 1, with equivalent elements similarly 
labeled. 

Generally speaking, system 100 includes at least platform-based 
imaging sensor 10 for generating perspective views, a platform-based 
processing system 102 and a platform-based data storage device 104. 

15 Typically, although not essentially, system 100 also includes various additional 
displays and controls, such as those illustrated in Figure 1 or others, to provide 
corresponding or additional functionality for camera function control (pan, 
zoom etc.), target designation and other input or output functions. 

The basic operation of system 100 and the corresponding method are 

20 illustrated in Figure 5. First, in step 110, a reference perspective view (e.g. 
view 42 above) together with a corresponding set of parameters defining a 
primary mapping transformation are stored in data storage device 104. The 
reference perspective view must at least partially overlap a current or intended 
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Field of view of imaging sensor 10. The primary mapping transformation T| 
corresponds to a mapping between the orthogonal photographic representation 
(e.g. representation 44 above) and the reference perspective image. At step 112, 
a current perspective view is obtained from imaging sensor 10. Processing 
5 system 102 is then used at step 114 to correlate the current perspective view 
(e.g. view 40 above) with the reference perspective view so as to generate a 
secondary mapping transformation T 2 between the current perspective view and 
the reference perspective view. The combination of the parameters defining T! 
together with the calculated transformation T 2 allow processing system 102 to 

10 map equivalent points in at least one, and preferably both directions, between 
the current perspective view and the orthogonal photographic representation 
(step 116), thereby enabling processing system 102 to convert between N-S 
coordinates and image pixels in either direction. 

It will immediately be appreciated that the present invention provides 

15 the platform with a high degree of autonomy and greatly reduces the data 
transfer requirements compared to the system of Figure 1. Specifically, 
continuous download of images from platform 16 to a base unit 18 is avoided, 
instead allowing concise communication between platform 16 and other 
platforms or base units in terms of coordinates within the orthogonal 

20 photographic representation or absolute geographic coordinates. Furthermore, 
the computational burden on the base unit is greatly reduced, freeing up its 
resources for alternative tasks and/or allowing a single base unit to manage 
many more platfocms. These and other advantages of the system and method of 

12 
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the present invention will be better understood in view of the following detailed 
description. 

Before proceeding further, it will be useful to define certain terminology 
as used herein in the description and claims. Firstly, the word "perspective" is 
5 used herein' to refer to any non-orthogonal viewing direction. The phrase 
"different perspective views" is used to refer to two views which have non- 
coincident viewing directions. The phrase "current perspective view" is used to 
refer to a perspective view currently being processed, and preferably, the view 
most recently input from imaging sensor 10. 

10 In a further issue of terminology, reference is made herein in the 

description and claims to an "orthogonal photographic representation". The 
term "orthogonal" is used herein to refer to a real or processed image which has 
a vertical effective viewing direction over its entire area. Satellite images may 
closely approximate to such conditions. Alternatively, other aerial images may 

15 be combined by image processing to construct the required view. This 
technology is well known in the art and will not be described further herein. 
Most preferably, the orthogonal photographic representations used by the 
present invention are associated with a geographic coordinate system so that 
the mapping transformations link locations within the current perspective view 

20 directly to geographical locations. 

The term "photographic representation" is used herein to refer to a real 
or simulated image which simulates the properties of an image generated by a 
corresponding type of imaging sensor. Thus, the photographic representation 

13 
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typically approximates to an aerial photograph taken at visible wavelengths. In 
certain cases, an orthogonal photographic representation approximating to a 
thermal aerial photograph may be used. The platform of the present invention 
may employ different , types of imaging sensors 10, and the photographic 
5 representation need not be of the same image type. 

The term "platform" is used herein in the description and claims to refer 
to any platform from which a perspective view of a region of interest can be 
obtained. Although illustrated herein in a preferred implementation on an 
airborne platform, it should be noted that the present invention is not limited to 

10 airborne platforms, and can equally be used where an imaging sensor is located 
on other platforms such as observation towers or look-out points. It should also 
be noted that the platform need not be a manned platform. In the case of an 
unmanned platform, such as an unmanned aerial vehicle ("UAV"), the user 
displays and input device may be omitted, or may be located at the 

1 5 corresponding base unit. 

The term "remote" is used herein to refer to any location not on the 
platform. Typically, although not necessarily, the remote components of the 
present invention are located at a base unit in real-time wireless communication 
with the platform. 

20 The system and method of the present invention are described as 

"pseudo-autonomous" in the sense that they operate within certain bounds in an 
autonomous manner to derive mappings between a current perspective view 

14 
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and an orthogonal reference while still being dependent upon an external 
source of data for intermittent updates, as will be described further below. 

Turning now to the features of the present invention in more detail, it 
should be noted that the reference perspective views may originate from any 
5 source including, but not limited to, imaging sensor 10. Thus, by way of 
example, for a fixed lookout position, one or more reference views may be 
generated during an initial one-time setup procedure and the primary 
transformation T { for each calculated. These views may be generated by any 
suitable imaging sensor located at the lookout position or nearby, and need not 

10 be generated by imaging sensor 10 used subsequently for implementation of 
the present invention. Similarly, for an airborne platform, the reference 
perspective views may optionally be obtained from a first platform and then 
subsequently used by a second platform to provide pseudo-autonomous 
operation according to the teachings of the present invention. By pre-loading a 

15 library of different views and their corresponding pre-calculated mapping 
transformations, it is possible to extend the pseudo-autonomous operation of 
the platform to substantially eliminate the need for image download and 
processing during operation. Switching between pre-loaded reference views 
may be performed automatically based upon various parameters such as: the 

20 current geographical position of the platform as derived from GPS or from the 
image correlation itself, the current attitude of the platform and viewing angles 
of the imaging sensor, and/or by the direction of exit of the FOV from the 
previously-used reference image. 

15 
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Preferably, the reference perspective view has a FOV significantly 
greater than the current perspective view. Specifically, the current perspective 
view preferably has a FOV corresponding to less than ten percent of the FOV 
of the reference perspective view, and more preferably less than three percent. 

5 This allows free movement of the imaging sensor FOV over a relatively large 
region of interest without requiring replacement of the reference view. 

Turning now to Figure 6, this shows a preferred implementation of the 
method of the present/invention where the reference image is generated, and 
periodically updated as needed, by use of imaging sensor 10. In this case, step 

10 110 is performed by initially deriving and storing the reference perspective 
view from imaging sensor 10 (step 120) and downloading the reference view to 
the remote processing system, identified by numeral 24 in Figure 4 (step 122). 
The intensive calculation required to generate primary mapping transformation 
T! is then performed by remote processing system 24 by use of the large 

15 database containing orthogonal photographic representation reference data in 
remote data storage device 30 (step 124). The parameters of the registration 
matrix are uploaded to the platform and stored in data storage device 104 (step 
126). The system and method are then ready to proceed with steps 112, 114 and 
116 as before. 

20 According to a further preferred feature of the present invention 

illustrated here, the system evaluates a mismatch criterion between the current 
perspective view and the reference perspective view (step 128). The mismatch 
criterion tests whether a new reference perspective view should be obtained. In 

16 
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a simple implementation, the mismatch criterion may simply be a failure to 
achieve sufficient correlation between the current perspective view and the 
reference perspective view to establish reliable registration between them. 
Alternatively, the mismatch criterion may be a numerical calculation indicative 

5 of poor correlation, or of large angular discrepancy or incomplete FOV overlap 
between the images, to allow preemptive replacement of the reference image to 
avoid loss of registration. When the mismatch criterion evaluation indicates 
that the reference image should be replaced, the system preferably returns to 
step 120 to generate (or acquire) a new reference perspective view and the 

10 corresponding primary mapping transformation. Most preferably, during the 
remote processing to derive the new primary mapping transformation 
parameters, the platform-based system continues to function using the previous 
reference perspective view, only replacing it with the new reference perspective 
view when the corresponding transformation parameters are received. In this 

15 manner, seamless continuous operation is ensured. 

It will be appreciated that the above descriptions are intended only to 
serve as examples, and that many other embodiments, are possible within the 
scope of the present invention as defined in the appended claims. 



17 
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WHAT IS CLAIMED IS: 

1. A method for providing a platform with pseudo-autonomous 
correlation between a perspective view generated by a platform-based imaging 
sensor and an orthogonal photographic representation, the method comprising: 

(a) storing on the platform: 

- - ■ - (i) - a reference perspective view at least partially overlapping - 
a current field of view of the imaging sensor, and 
(ii) parameters of a primary mapping transformation 
corresponding to a mapping between the orthogonal 
photographic representation and said reference perspective 
image; 

(b) obtaining a current perspective view from said imaging sensor; 
and 

(c) employing a processing system located on the platform to: 

(i) correlate said current perspective view with said reference 
perspective view so as to generate a secondary mapping 
transformation between said current perspective view and 
said reference perspective view, and 

(ii) use said primary and secondary mapping transformations 
to map equivalent points in at least one direction between 
said current perspective view and said orthogonal 
photographic representation. 
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2. The method of claim 1, wherein said current perspective view has 
a field of view corresponding to less than ten percent of an area of a field of 
view of said reference perspective view. 

3. " The method of claim 1, wherein said reference perspective view 
is derived from said imaging sensor, and wherein said parameters of a primary 
mapping transformation are derived by: 

(a) downloading said reference perspective view from the platform 
to a remote processing system; 

(b) employing said remote processing system to correlate said 
reference perspective view with the orthogonal photographic 
representation so as to generate said parameters; and 

(c) uploading said parameters from the remote processing system to 
the platform. 

4. The method of claim 3, further comprising: 

(a) evaluating a mismatch criterion between said current perspective 
view and said reference perspective view; and 

(b) conditional upon a result of said mismatch criterion evaluation, 
employing said imaging sensor to generate a new reference 
perspective view and repeating said steps of: downloading; 
employing said remote processing system; and uploading. 
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5;. The method of claim 1, wherein the platform is an' airborne 
platform. 

6. The methpd of claim 1, wherein the orthogonal photographic 
representation has a defined geographic coordinate system. 

7. The method of claim 1, wherein said reference perspective view 
is derived from a source other than said imaging sensor, the platform being an 
airborne platform, said step of storing being performed prior to launch of said 
platform. 

8. The method of claim^l, wherein said storing includes storing a 
plurality of said reference perspective views and a plurality of sets of 
corresponding parameters of the primary mapping transformation, the method 
further comprising selecting one of said plurality of said reference perspective 
views at least partially overlapping the current field of view of the imaging 
sensor for correlating with said current perspective view. 

9. An image correlation system for providing pseudo-autonomous 
platform-based correlation between a perspective view generated by a 
platform-based imaging sensor and an orthogonal photographic representation, 
the system comprising: 

(a) a platform-based imaging sensor for generating perspective 
view's; 

20 
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a platform-based data storage device for storing: 

(i) a reference perspective view, and 

(ii) parameters of a primary mapping transformation 
corresponding to a mapping between the orthogonal 
photographic representation and said reference perspective 
image; and 

a platform-based processing system associated with said imaging 
sensor aqd with said data storage device, said processing system 
being configured for: 

(i) correlating a current perspective view derived from said 
imaging sensor with said reference perspective view so as 
to generate a secondary mapping transformation between 
said current perspective view and said reference 
perspective view, and 

(ii) employing said primary and secondary mapping 
transformations to map equivalent points in at least one 
direction between said current perspective view and the 
orthogonal photographic representation. 
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10. " The system of claim 9, further comprising: 

(a) a remote processing system including a remote data storage 
device storing the orthogonal photographic representation; and 

(b) . a communication link between said platform-based processing 

system and said remote processing system, 
wherein said remote processing system is configured to: receive an image 
downloaded from said imaging sensor via said communication link, said image 
being designated as a reference perspective view; correlate, said reference 
perspective view with the orthogonal photographic representation so as to 
generate said parameters; and upload said parameters via said communication 
link to said platform-based data storage device. 

11. The system of claim 10, wherein said communication link is a 
wireless communication link. 

12. The system of claim 9, wherein the platform is an airborne 
platform. 

13. The system of claim 9, wherein the orthogonal photographic 
representation has a defined geographic coordinate system. 

14. The system of claim 9, wherein said platform-based data storage 
device is configured to store a plurality of said reference perspective views and 
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a plurality of sets of corresponding parameters of the primary' mapping 
transformation, said platform-based processing system being further configured 
to select one of said plurality of said reference perspective views for correlating 
with said current perspective view. 
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